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(57) ABSTRACT 

An automated system for controlling multiple pan/tilt/zoom 
video cameras in such a way as to allow a person to be 
initially designated and tracked thereafter as he/she moves 
through the various camera fields of view. Tracking is 
initiated either by manual selection of the designated person 
on the system monitor (82) through the usage of a pointing 
device (83), or by automated selection of the designated 
person using software. The computation of the motion 
control signal is performed on a computer (81) through 
software using information derived from the cameras con- 
nected to the system, and is configured in such a way as to 
allow the system to pass tracking control from one camera 
to the next, as the designated person moves from one region 
to another. The system self-configuration is accomplished by 
the user's performance of a specific procedure involving the 
movement and tracking of a marker throughout the facility. 
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FIG. 2 Typical Guard Monitoring System 
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FIG. 3 Automated Video Person Tracking System 
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FIG. 7 Flowchart of Program in Regular Operation 
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FIG. 8 Flowchart of Program in Configuration Mode 
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AUTOMATED VIDEO PERSON TRACKING limited to extremely high contrast situations where the target 

SYSTEM is extremely visually distinct from the background, due to 

the simplicity of the tracking algorithms. 

BACKGROUND OF THE INVENTION More recently, U.S. Pat. No. 5,434,617 to Bianchi (1995) 

„, . . i . . 'j mi 5 combines a fixed camera and a PTZ camera to track an 

Tta invention relates to video surveillance systems, individual, v^m the tracking algorithm based on pixel dif- 

specificaUy to a system that will automatically track a person ferencing techniques calculated on the fixed camera view, 

as he/she moves around. ^ a resu i ting mo tion control signal used to point the PTZ 

Many retail stores use video surveillance of customers to camera. This system requires that a fixed camera always be 

prevent shoplifting. Two types of cameras are used for this 30 used in conjunction with the PTZ, and is once again limited 

purpose: fixed cameras, and. pan/tilt/zoom cameras (PTZ*s). to a single PTZ camera's field of view. The system does have 

The fixed cameras are stationary, and are used to constantly a slightly more robust tracking algorithm than those of 

monitor the same location. The PTZ cameras can be rotated Saccomani and Sacks, but will still not hold up against a 

around, and are used to zoom in on a particular location or complex background. 

person. Stores generally have a mixture of both types of 0ther recent patents (e.g. U.S. Pat. Nos. 5,552,823 
cameras mounted on the ceiling throughout the store, with 15 ( 1996 )> 5,706,362 (1998), 5,714,999 (1998), and 5,754,225 
the resulting footage from these cameras being recorded for { l99S )) ^similar in that they disclose systems implement- 
potential use as future evidence. in S al g° nthms for ^cking individuals with cameras, with 

.. varying means of designating targets- and varying 

The field of view of a fixed camera is constant, so it is complexity/robustness levels of the algorithms. All of these 

usually set up to cover a broad area, or one of particular 20 prior art systems m limited to single camera usage, and do 

interest (e.g. a jewelry counter or cash register). If it is not discuss the issues arising from multiple camera usage 

covering a broad area, such as the whole music department, with hand-offs and self-configuration, that are addressed by 

a single person will only comprise a small part of the entire the current invention. 

image. This can make it difficult to determine from the Some computer automated tracking systems for outdoor 

image exactly what a person is doing. It might be impossible 25 vehicle tracking (e.g. in large facility yards) are currently 

to distinguish a shoplifting incident, such as a person putting offered for sale commercially, such as the e. clips™ system 

a CD into a pocket, from a normal gesture, such as some- from Cortex, Inc. This system is limited to controlling a 

one's sticking his hand into his jacket pocket. single PTZ camera as well, however, and does not perform 

The limitations of fixed cameras give rise to the use of ' he or auto-configuration procedures necessary 

PTZ cameras. If the camera is pointed and zoomed in 30 iox ^ wth multiple < f mera * t , ff . 

. t , i i j . j ■» There are no systems that are currently offered 

appropriately, a person can be observed in more detail, so . „ j i * *l * c / *• n 

*l . • / !■ i j • * ■ • . . c . r V- commercially, and none known in the art, for automatically 

that innocent actons can be distinguished from theft. For , rackin individuals usi multi le cameras as „, mo / e 

this reason, it is common practice for secuntypersonnel to throughout storeSj „ disc i osed by this invention. 

monitor customers by following them with PTZ s as they 

move around the store. 35 BRIEF SUMMARY OF THE INVENTION 

In the prior art, the PTZ's are controlled with joysticks or Thus, the present invention, with computer automated 

a mouse and computer interface, with the guard being tracking of individuals, has several advantages over the 

required to manually follow a customer with the camera as current manual tracking methods. Specifically, it: 

they move about. This manual tracking process has the 1) allows the guard to concentrate on other activities, as 

following disadvantages: 40 necessary, after the initial designation of the subject to 

1) the guard needs to devote his/her full attention to the be tracked; 

tracking process; ' allows more than one subject to be tracked 

«v . . . . . , j | simultaneously, subject to the positions of the subjects 

2) it is impossible for a single guard to track more than ■ f , ' - J 3V3ll;ihlp * rw 



and the number of available cameras; 
45 3) results in smoother and more complete video coverage; 



one subject simultaneously; 

3) limitations in human manual dexterity make resulting ..w . . ir 
video coverage jerky and incomplete; 4 > ^ a object continuously for as long as necessary, 

JX & J 3 . v ' . „ untri the subject leaves the store; 

4) limited human attention spans can result in a guard s c\ n i c ty^j 

7 . . . 4 t . ,. \ , • * * i u- 5) allows seamless shifting from one PTZ camera to a 

losmg interest in a subject, and ceasing to track him, , 4 . ,r 4 . . 

, c & iL ,. , J a , & different one, as the subject moves from one region in 

before the subject has left the store; 50 ^ stQre tQ aQ0ther; 

5) the process of shifting from one PTZ camera to a 6) does M mattef if ^ rf becomes mvoluntaril 
different one, as the subject moves from one region in attracted temporarily, since the system will continue to 
the store to another, is a difficult process to do manually ' tfaclc automat f ca ii y 

and usually results in a gap in coverage; and Fimher objects ^ advantages indude allowing mlegra . 

6) if the guard is involuntarily distracted, e.g. with a 55 tion 0 f the tracking function with other computer 
phone call, there will be a gap in coverage. automation, so that, for instance, a database of known 

The area of tracking moving objects with cameras has a previous shoplifters may be maintained, and enabling selec- 
long history, with origins in military applications. U.S. Pat. t j on 0 f potential tracking targets automatically through the 
No. 3,953,669 to Saccomani (1976) discloses the selection use of incorporated face recognition software. Still further 
of a target with a fight pen, with a relatively simple proce- 60 objects and advantages will become apparent from a con- 
dure for producing an error signal allowing the window to be sideration of the following detailed description, the accom- 
moved to keep the target centered. U.S. Pat. No. 3,988,534 pan ying drawings, and the claims, 
to Sacks (1976) discloses another system with an automati- 
cally generated "tracking gate" target selection, with another BRIEF DESCRIPTION OF THE SEVERAL 
procedure for producing an error signal that controls a servo 65 VIEWS OF THE DRAWING 
allowing the camera to keep the target centered. Both of FIG. 1 shows the basic layout of a typical store with a 
these systems are for use with single cameras only, and are multi-camera security system. 
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FIG. 2 shows a basic guard monitoring station. 

FIG. 3 shows a block diagram of the invention and its 
connections to the other system components. 

FIG. 4 shows typical camera positions as someone enters 
the store. 

FIG. 5 shows camera positions for a "hand-off" as the 
shopper moves through the store. 

FIG. 6 shows camera positions and guard path for the 
initial camera configuration procedure to determine fields- 
of-view. 

FIG. 7 shows a flow diagram for a particular embodiment 
of the software during regular tracking operation. 

FIG. 8 shows a flow diagram for a particular embodiment 
of the software during the initial configuration of the system. 

REFERENCE NUMERALS IN DRAWINGS 
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DETAILED DESCRIPTION OF THE 
INVENTION 

The preferred embodiment of the present invention is 
illustrated by way of example in FIGS. 1-8. 

The environment in which a typical embodiment of the 
invention might be used is shown in FIG. 1 and FIG. 2. The 
automated tracking system 80 is located in a central moni- 
toring station, being connected by cables 120 to the store 
PTZ cameras 30, 31, the PTZ joystick controller 60, and the 
monitors 50 and 51. The total possible fields of view of 
cameras One and Two 3031 are shown by the two circles 
100,101. It is apparent that the store may be divided into 
three regions: one region which may be viewed only with 
PTZ camera One 30, another region which may be viewed 
only by PTZ camera Two 31 and a region between them 103 
viewable by both cameras 3031. It should be noted that the 
choice of two cameras 3031 in this system is arbitrary — the 
tracking system will also work in a store network with many 
more cameras than just two. 

The central monitoring station is shown in FIG. 2. The 
tracking system is activated by the guard 70 who uses the 
mouse 83 to click on a person 20 (the shopper) in the field 
of view 100 of PTZ camera One 30, whose image is shown 
on both the tracking system monitor 82 and the standard 
monitor 50 for PTZ camera One 30. The tracking system 80 
then takes over control of the panning and zooming func- 
tions of the PTZ camera One 30, automatically moving and 
focusing it so as to keep the shopper 20 in constant view of 
the camera. 

The components of the invention itself are shown in FIG. 
3. The heart of the system is the computer 81, which has the 
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standard additions of a mouse 83, keyboard 84, and monitor 
82. The computer 81 also has several interfaces connected to 
it, either through ports or as cards plugged into the comput- 
er's data bus system. Among these are a joystick-input 
interface 85, a PTZ control interface 86, and a camera 
digitizer 88. All of these components are well known and 
may be accomplished by any compatible system of compo- 
nents now known or hereafter developed. 

The joystick-input interface 85 receives an input signal 
from the joystick 60, and converts this to an input to the 
computer 81. The camera digitizer 88 and the PTZ control 
interface 86 are connected to the PTZ Control Multiplexer 
87, which is used to pass through the image and control 
signals from the appropriate camera, out of all the cameras 
30,31 connected to it. The camera selection and motion 
control signals are signals originating in the computer 81, 
and sent out through the PTZ control interface 86, to the PTZ 
control Multiplexer 87. The motion control signal is then 
passed through the PTZ control Multiplexer 87 to the 
currently selected PTZ camera 30 or 31. The input video 
signal from the selected PTZ camera 30 or 31 is passed back 
through the PTZ control Multiplexer 87 at the same time, 
and is digitized by the Camera Digitizer 88, with that 
digitized form of the image sent back to the computer 81. In 
the event that the guard wants to manually use a PTZ camera 
not currently used by the tracking system, another set of 
selection and motion control signals originating in the 
computer 81 will be sent out through the PTZ control 
interface 86, to the PTZ control Multiplexer 87. This manual 
motion control signal will then also be passed through the 
PTZ control Multiplexer 87 to the another manually selected 
and controlled PTZ camera 30 or 31. 

The motion control signal sent out to camera 30 or 31 for 
automatic tracking of the designated shopper 20 will be 
determined by the computer 81 from tracking software 
continuously processing the digitized image from camera 
digitizer 88. 

A description of a particular tracking algorithm of the 
preferred embodiment of the invention is as follows. The 
initial view is guaranteed to have the designated shopper 20 
to be tracked in it, because the guard 70 selects this 
individual by clicking on him/her in the image, as shown in 
FIG. 4. The movement of the shopper 20 is used to segment, 
this object from the other objects in the image, and a color 
histogram of the shopper can be formed at that point. This 
color histogram information,' along with last known 
position, estimated new position based on velocity, height, 
width and overall size, are used to perform matches to this 
object in subsequent images. The zoom factor is controlled 
by zooming in to the point where the shopper's image fills 
a reasonably large part of the image, but does not exceed it's 
boundaries. The centroid of the shopper 20 is calculated, and 
motion control signals sent to the FTZ camera 30 or 31 to 
ensure that the camera moves to maintain that shopper's 
centroid near the center of the image. If the shopper 20 
temporarily becomes obscured by some other object (e.g. a 
high display shelf), the software will zoom out slightly and 
search in the last known vicinity of the shopper until 
tracking is reestablished. 

As the shopper 20 moves throughout the store, it is 
necessary to "hand -off" the shopper from one camera to 
another 30 or 31, as required in order to maintain a good 
view of the shopper. Through the configuration process 
below, the tracking system, computer 81 has a record of 
where all the various FYZ cameras 3031 in the store are 
located, and of what their total fields of view 100, 101 are. 
When the shopper 20 reaches the designated boundary for a 
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particular camera 30 or 31, the tracking system automati- 
cally moves and focuses the next appropriate camera 30 or 
31 to cover that position, which is si ill known to the system. 
This camera is then selected for auto-tracking control by the 
system, and the tracking procedure described above contin- s 
ues in the new field of view 100 or 101. This is depicted in 
FIG. 5. 

The configuration process is necessary in order to deter- 
mine the total fields of view 100,101 that the various 
cameras 30,31 in the store system have. This geometry will 1Q 
vary from store to store, and. must be determined 
automatically, since it will not be reasonable to expect the 
user to modify the program by entering camera coordinates 
and field of view angles. In order to accomplish this, a 
special installation/configuration program will be run when 
the system is first installed, with all the cameras 30,31 15 
looking for a special flashing marker as a guard slowly traces 
out a predetermined path throughout the store. This process 
is shown schematically in FIG, 6. This allows the software 
to determine relative camera positions with respect to each 
other, and corresponding boundaries of fields of view 100, 20 
101. 

Flow diagrams of a preferred embodiment of this tracking 
software are included as part of this document in FIG. 7 and 
FIG. 8. The software may be written in Visual C++™ for a 
computer 81 running the Windows NT™ operating system. 
It should be noted that many variations in the algorithm are 
possible, with accordingly varying levels of system tracking 
performance. 

The tracking system of the present invention is operated 3Q 
in the following manner: 

1) the guard 70 selects a PTZ camera 30 or 31 and zooms 
in on a particular shopper 20, as represented in FIG. 4; 

2) the guard moves the mouse 83 until the mouse cursor 

is superimposed on the shopper's image displayed on 35 
the monitor 82, and clicks; 

3) the tracking system 80 at that point takes over control 
of whatever PTZ cameras 30 or 31 are necessary in 
order to maintain continuous tracking of the subject; 

4) if the guard 20 wants to manually control another PTZ 40 
camera 30 or 31 that is not in use by the tracking system 
80, he/she selects another camera using the program's 
graphical user interface, and uses the joystick 60 to 
control that camera. 

If the shopper 20 moves out of one camera's field of view 45 
100 to another 101, it will be necessary for the tracking 
system to perform the hand-off described above; that is, the 
tracking system will determine which camera 30 or 31 is 
appropriate for tracking the shopper, and transfer automated 
tracking control to that camera. This occurs automatically 50 
through usage of the configuration information, without 
requiring the guard's 20 intervention. 

Due to the variable geometry of camera placement in 
stores, an initial configuration/installation procedure is 
necessary, in order for the system to know which camera 30 55 
or 31 to use to cover a particular area of the store. This is 
accomplished by an initial configuration procedure depicted 
in FIG. 6, where a special installation/configuration program 
is run while the guard 20 walks slowly along a predeter- 
mined path with a special flashing marker 160 on a pole, 60 
while all the cameras are swept through their total fields of 
view 100 and 101 recording the position of the marker 160. 
This allows the tracking system 80 to determine on it's own 
what the fields of view of the various cameras are, without 
human intervention. This is an important practical consid- 65 
eralion for the users of the tracking system of the present 
invention. 



Accordingly, the automated video person tracking system 
of this invention provides the user with numerous advan- 
tages over the current prior art methods used, while being 
easy to install and operate. The system automates a task that 
is ill suited for accomplishment by humans, because of 
inherent limitations in both dexterity and attention span. The 
system also provides smoother and more continuous video 
coverage than a human could in most cases, and frees the 
user to perform other tasks. Awkward processes, such as the 
hand-off, or waiting for someone to emerge from behind an 
obstruction, are accomplished automatically, and the user is 
able to monitor and control other cameras while the tracking 
process is taking place. 

Although the description above contains many 
specificities, these should not be construed as limiting the 
scope of the invention but as merely providing illustrations 
of some of the presently preferred embodiments of this 
invention. For example, different algorithms might be used 
to accomplish the tracking; the environment that the inven- 
tion is used in might be someplace other than a store; the 
interfaces mentioned above might be combined into a single 
unit, etc. 

Thus the scope of the invention should be determined by 
the appended claims and their legal equivalents, rather than 
by the examples given. 

I claim: 

1. A system for automated tracking of persons with 
pan/tilt/zoom cameras, comprising: 

(a) a computer system, 

(b) a pointing device connected to said computer system, 
providing a means of designating a person to be 
tracked, 

(c) a means of digitizing an input camera video signal 
connected to said computer system, 

(d) an interface means allowing a camera motion control 
signal to be input to said computer system, 

(e) a camera multiplexing means allowing a particular set 
of camera video and motion control signals to be 
selected from a multiplicity of said signals and passed 
through to appropriate input and output connections, 

(f) an interface means allowing a selector signal and 
camera motion control signals to be output from said 
computer system to said camera multiplexing means; 
and 

(g) a software program running on said computer imple- 
menting an algorithm providing a means of issuing 
camera motion control signals to said interface means 
and multiplexing means in such a manner as to keep the 
designated person to be tracked centered in the field of 
view of a pan/tilt/zoom camera. 

2. The system of claim 1 wherein said pointing device is 
a mouse. 

3. The system of claim 1 wherein said camera multiplex- 
ing means and interface means are combined in a single unit. 

4. The system of claim 1 wherein the designation of the 
person to be tracked is performed automatically by a soft- 
ware program. 

5. The system of claim 4 wherein the software program's 
designation of the person to be tracked is based on an 
automated face recognition of the person to be tracked. 

6. The system of claim 1 wherein said software program 
determines the necessary information about camera posi- 
tions and fields of view through an automatic self- 
configuration procedure. 

7. The system of claim 6 wherein said automatic self- 
configuration procedure is facilitated by the movement 
through the facility of a special, visibly distinct marker. 
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8. A method for producing camera motion control signals 
from a computer system in such a manner as to keep a 
designated person tracked in the view of a camera as that 
designated person moves about, comprising: 

(a) designating the person to be tracked in an image, 

(a) segmenting out the designated person being tracked 
from the rest of the image using motion-based 
techniques, 

(b) determining various characteristics of the segmented 
out region of the image, including color histogram 
information, and height, width, and overall area in 
pixels, 

(c) acquiring another image a short time later from the 
camera, 

(d) matching the various characteristics of the designated 
person with those of all the moving segmented out 
objects in the new image, in order to find a best match; 



9. The method of claim 8 wherein the step of determining 
said characteristics of the segmented out region of the image 
includes height, width, and overall area in pixels. 

10. The method of claim 8 wherein the step of determin- 
ing said characteristics of the segmented out region includes 
the last position of the region in the overall image, and 
estimating the next likely position of the region based on the 
last computed velocity. 

11. The method of claim 8 further comprising the step of 
providing a self-configuration procedure whereby the nec- 
essary information about camera positions and fields of view 
is determined. 

12. The method of claim 11 wherein said self- 
is configuration procedure further includes the step of the 

movement through the facility of a special, visibly distinct 
marker. 

13. The method of claim 8 wherein said step of designa- 
tion of a person to be tracked further includes the step of 
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and 

(f) computing the appropriate direction to move the 20 using face recognition software, 
camera, in order to keep the best match object approxi- 
mately centered in the camera view. * * * * 
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